Although narratives on patient forums are a valuable source of medical information, their systematic detection and analysis has so far been limited to a single study. In this study, we examine whether psycholinguistic features or document embeddings can aid identi cation of narratives. We also investigate which features distinguish narratives from other social media posts. This study is the rst to automatically identify the topics discussed in narratives on a patient forum. Our results show that for classifying narratives, character 3-grams outperform psycho-linguistic features and document embeddings. We found that narratives are characterized by the use of past tense, health-related words and rst-person pronouns, whereas non-narrative text is associated with the future tense, emotional support words and second-person pronouns. Topic analysis of the patient narratives uncovered fourteen di erent medical topics, ranging from tumor surgery to side e ects. Future work will use these methods to extract experiential patient knowledge from social media.
individual terms, which may overlap signi cantly between narratives and factual statements about the same topic. Secondly, we explore how narratives di er from other types of posts by studying which features are in uential in identifying narratives and which posts are classi ed incorrectly. Thirdly, we analyze how prevalent narratives are on a cancer patient forum and which topics these narratives discuss. 2
Narratives on patient forums have mainly been studied qualitatively (e.g. [vUKDT+09]). The automatic identication of narratives on a patient forum is limited to the study by Verberne et al. [VBSEng] on a Dutch cancer forum. They identi ed narratives with a F1 of 0.911 using only the lower-cased words of the posts as features. They also found that various linguistic factors (1st person singular, 3rd person and negations) and psychological processes (social processes and religion) were correlated with the presence of narratives. These psycho-linguistic features were measured using the Linguistic Inquiry and Word Count (LIWC) method [TP10].
Additionally, research into self-reported adverse drug responses (ADRs) has led to the development of classi ers for di erentiating between factual statements of ADRs and personal experiences of ADRs on social media [BY12, NSO+15, SG15]. However, these classi ers are highly speci c and thus not suitable for identifying patient narratives in general.
Another closely related eld is the classi cation of personal health mentions on social media i.e. posts that mention a person who is a ected as well as their speci c condition, such as: `my granddad has Alzheimer's'. Presently, only two studies have investigated this task. The rst by Lamb et al. [LPD13] focused on separating u awareness from actual u reports on social media. More recently, Karisani et al. [KA18] introduced WESPAD, a classi er for personal health mentions, which attains state-of-the-art performance for seven di erent health domains including stroke, depression and u infection. Nonetheless, a personal health mention alone is not su cient to consider the post a narrative, and thus these classi ers are also inadequate for our purpose. 3 3.1
Our data consists of an open, international Facebook forum for patients with Gastrointestinal Stromal Tumor (GIST)1. It is moderated by GIST Support International and consists of 36,722 posts with a median length of 20 tokens.
The data was lowercased and tokenized with NLTK. Due to the noisy nature of user-generated content, especially in the spelling of medical terms, we applied a tailored preprocessing pipeline2 to our data. Firstly, an existing normalization pipeline for social media [Sar17]3 was used to normalize tokens to American English and to expand generic abbreviations used on social media. Hereafter, domain-speci c abbreviations were expanded with a lexicon of 42 non-ambiguous abbreviations, generated based on 1000 posts and annotated by a domain expert and the rst author. Spelling mistakes were detected using a combination of relative frequency and edit distance to possible candidates and corrected using weighted Levenshtein distance. Correction candidates were derived from the corpus itself. Drug names were normalized using the RxNorm database [Nat]. Non-English posts were removed using langid [LB12]. Punctuation was removed, but stop words were not, as we expect function words to play a role in the expression of narratives. 3.3 3.3.1
We randomly selected 1050 posts for annotation. The annotators were asked to indicate per message whether it contains a personal experience. They were not provided with its context. Personal experiences did not need to be about the author but could be about someone else. This de nition was based on earlier work by Verberne et al. [VBSEng] and van Uden-Kraan et al. [vUKDT+08]. The rst 50 posts were annotated individually by the 1https://www.facebook.com/groups/gistsupport/ 2The preprocessing scripts can be found at: https://github.com/AnneDirkson/LexNorm 3https://bitbucket.org/asarker/simplenormalizerscripts rst author and another PhD student to improve the annotation guidelines.4 The remaining 1000 posts were divided equally into six sets of 200 posts, with 40 posts (20%) overlapping between all sets. The overlap was used to calculate the pairwise Cohen's kappa. There were seven annotators in total: six PhD students and one GIST patient. Each sample was assigned to an annotator, apart from one sample which was divided between two PhD students. To be able to include the overlapping sample in the classi cation, we opted to use the annotations of the GIST patient for these 40 posts.5 3.3.2
Four feature sets were derived from the text data: word unigrams, character n-grams (using the CountVectorizer function in sklearn), psycho-linguistic features, and document embeddings. For both word unigrams and character n-grams, we investigated whether TF-IDF weighting would improve performance compared to raw counts. Additionally, we explored whether stemming or lemmatising the data prior to extracting the unigrams could improve performance. Psycho-linguistic features were based on the LIWC 2015 [TP10]. Punctuation categories were discarded, resulting in 82 LIWC features in total. LIWC is a well-known method for investigating psychological processes in text and includes both linguistic (e.g. rst-person pronouns) and psychological categories (e.g. positive emotions). The last feature set consisted of document embeddings: a doc2vec model [LM14] was trained on the labeled training data for each fold in the cross-validation. We combine a distributed memory model with a distributed bag of words model, as recommended by Le and Mikolov [LM14]. We also attempted to train document embeddings rst on the unsupervised data and then re-train on the supervised data, but this led to nonsensical classi cation features. 3.3.3
Classi ers were evaluated separately for each feature set. We ignored all posts that had been left empty by the annotator (the annotator chose neither yes nor no): three posts were ignored for this reason. For word unigrams, character n-grams and psycho-linguistic features, we compared four sklearn classi cation algorithms: Multinomial Naive Bayes (MNB), linear Support Vector Classi cation (LinearSVC), Stochastic Gradient Descent (SGD) with log loss, and K Nearest Neighbours (KNN). These were chosen according to the following criteria: (1) known to perform well on text data, (2) recommended for small data sets and (3) able to calculate probabilistic outcomes. The latter enabled us to use probabilistic ensembles. The doc2vec representations combined with Logistic Regression were used as classi er in itself: the document representations were tagged with the labels of the training data. This model was then used to derive vector representations for new documents. To test if a combination of feature types could improve performance, we evaluated soft voting (argmax of the sums of the predicted probabilities) of the best individual classi ers for the best performing variants of each feature set. Signi cance testing was done with pair-wise t-tests.
To evaluate the performance, the average F1 score of a 10-fold cross validation was used. For each run, hyper-parameters were tuned for that speci c training set using a 10-fold grid search on the training data. The tuning grids were based on sklearn documentation: C from 10-3 to 103 (steps of x10) for LinearSVC and Logistic Regression; number of neighbors from 3 to 11 (steps of 2) for KNN; and max iterations from 2 to 2048 (steps of x2) and alpha from 10-8 to 10-2 (steps of x10) for SGD. The dimensionality of the document vectors was tuned with a grid of 100 to 400 (steps of 100). 3.4
To label the remaining data, the best performing classi er was used with the hyper-parameter settings that were optimal in the majority of the training sets. To investigate which topics are discussed in the patient narratives, we used topic modelling with non-Negative Matrix Factorization of the TF-IDF weighted tokens without stopwords. Topic coherence, measured using TC-W2V [OGCC15], was used to select the number of topics. Topic labels were assigned manually by exploring the words with the highest weights and the top-ranked (i.e. most relevant) messages per topic.
4The annotation guidelines can be found at: https://github.com/AnneDirkson/NarrativeFilter 5The annotated data is available upon request in order to protect the privacy of the patients 4.1
The data was slightly imbalanced, with 37.7% of the posts containing a narrative, resulting in a majority baseline of roughly 0.62. The inter-annotator agreement was substantial ( = 0:69).
A Linear SVC on character 3-grams achieves the highest F1 score (Table 1), although character 4-grams (p = 0:526), stemmed unigrams (p = 0:930) and lemmatised unigrams (p = 0:587) do not perform signi cantly worse. Character 5- and 6- grams also do not perform worse overall (p = 0:122 and p = 0:169), but their recall is signi cantly lower (p = 0:023 and p = 0:029). The classi ers for the best performing document embeddings (DBOW+DM) and psycho-linguistic features, however, are signi cantly worse overall than character 3-grams (p = 0:0055 and p = 0:026 respectively). Employing TF-IDF weighting does not aid any of the unigram or character n-gram features. Additionally, neither feature selection (F1=0.761) nor word boundaries (F1=0.796) improve the performance of character 3-grams. Using a range of character n-grams, namely 3-to-4 (F1=0.814), 3-to-5 (F1=0.814), or 3-to-6 (F1=0.812), also does not boost performance.
Ensemble classi cation did not perform better than character 3-grams alone (see Table 2). Nevertheless, an ensemble of all four feature types is signi cantly more precise than all other classi ers (p = 0:0048 compared to the second best). To further explore why ensemble classi cation does not manage to improve overall performance, we investigated the predictions of individual classi ers. As can be seen in Table 3, there is a high degree of overlap between the predictions based on character 3-grams and the other feature sets (88.3%, 83.8% and 84.4% respectively). Consequently, the vast majority of the predictions cannot be improved by complementing character 3-grams with these feature sets. Interestingly, 4.7% of the posts are misclassi ed by all feature sets. Considering the non-overlapping predictions, the percentage of correct predictions was higher for character 3-grams than for either document embeddings or psycho-linguistic features in a pair-wise comparison. Thus, it appears that adding these features would be more detrimental than bene cial to narrative classi cation. you _yo ou_ ur_ our ere_ vher it _wi aare rr _ar a_he -n_so nfor o wil N_ma r_y pra ray ist ere group you join your happy e message
keep itv company -rraan bsuhptgltaroooartdneys n services o Ninteresting
free support find click posted Social Cognitive Processes Informal language Body Past Focus They She/He Causal Negative emotions Quantifiers Motion Health Reward Feel Time Prepositions Male Nonfluencies Biological processes Function words Assent Anger Authenticity −0.025 0.000 0.025 0.050
Weights (c) LIWC 0.0
0.5 Similarity Narratives are typically distinguished by terms relating to the past tense (was, had, years ), health (imatinib, tumor, surgeri ) and rst-person narrative (my, i ) (see Figure 1). This is corroborated by the character 3-grams, psycho-linguistic features and document embeddings. Some of the important terms for non-narrative texts are also health-related (patients, gist ) and rst-person narrative (we, us), which showcases the di culty of the task at hand. In general, non-narrative texts seem to focus more on emotional support (prayer, share, may ), secondperson narrative (you, your ) and the future (may, will ). The psycho-linguistic features additionally reveal that narratives contain more mentions of causality and negative emotions. In contrast, non-narrative texts seem to contain more positive emotions. Lastly, as predicted, function words appear important for classifying narratives in social media, and it is thus advisable to not remove stopwords. 4.4
Error analysis reveals that a signi cant proportion of the errors is due to incorrect annotation: 36.9% of the false positives and 36.2% of the false negatives were labelled incorrectly (see Table 4). Speci cally, annotators have di culty correctly labelling discussions about personal medical facts or side e ects as narratives (e.g `i have been on imatinib 5 months and lost 1/3 of my hair' ). Conversely, annotators may incorrectly judge posts that give emotional support, external information or advice to be narratives while they are not (e.g. `i may be wrong but total gastrectomy sounds very extreme for two small gist' ).
The incorrect labelling may have impacted the automated classi cation such that these categories are also more di cult for the computer to distinguish. The classi er does, however, appear to outperform human judgement and to some extent `correct' their mistakes. In fact, its performance may be underestimated by the metrics based on these incorrect labels. Other types of posts that appears challenging for the computer are posts that lack context or contain questions. The former are often answers to unknown questions posed earlier in the thread. The percentage of narratives in the unlabelled data is 37.0 %, which is comparable to the annotated sample. This results in a total of 13.436 posts for topic modelling.6 4.5.2
5
The TC-W2V metric [OGCC15] identi es the optimal number of topics to be fourteen. The resulting topics relate to di erent aspects of the medical process for GIST patients (see Table 5). Note that imatinib is the most commonly used medication.
The detection of narratives was most optimal when using character 3-grams. Their strength is in their ability to cluster relevant word types based on su xes and pre xes. This is especially relevant in the medical domain e.g. all cancer medication for GIST ends in `nib'. In contrast, psycho-linguistic features appear to su er from oversimpli cation, because they aggregate words that de ne di erent classes into one category e.g. we and my into the umbrella category of rst person pronouns (see Figure 1). The use of document embeddings may have been hampered by the small size of the data. An alternative explanation could be that incorrect labelling impacts these features more strongly than word-based features.
Narratives could be di erentiated most strongly by their use of past tense, rst-person narrative and healthrelated words. The rst two are in line with linguistic de nition of a narrative. The stronger focus on health, however, may indicate that patients prefer to share their own health experiences than health information from external sources.
Annotating narratives appears a challenging task, despite providing annotators with a guideline based on previous work [VBSEng] and validated through initial annotation by two annotators. This is underscored by our inter-annotator agreement ( = 0:69) which was comparable to that of Verberne et al. [VBSEng] ( = 0:71). Our classi er performed less well that their system (F1 = 0.91), which may be explained by their larger sample of annotated data (2.051 posts).
Inevitably, our results depend on the choice of what constitutes a narrative and how annotators interpret this de nition. It appears that especially the line between a medical fact about oneself and a medical experience is fuzzy for annotators. Future studies could perhaps use this knowledge to develop clearer guidelines.
6The code for unsupervised narrative ltering is shared at: https://github.com/AnneDirkson/NarrativeFilter
For the detection of patient narratives on social media, psycho-linguistic features and document embeddings are outperformed by character 3-grams. These narratives are associated with the past tense, health and rst-person pronouns, whereas non-narrative text is associated with the future tense, emotional support and second-person pronouns. The patient narratives could be subdivided into discussions of fourteen di erent medical topics, ranging from surgery to side e ects. Future work will develop automated methods for the extraction of patient knowledge from the narratives. 7
This work was nanced by the SIDN fonds. The authors also thank H. Vos, G. Wiggers, W. Verschoof, A. Brandsen, D. Gawehns, P. Dhar, M. Vinkenoog and G. van Oortmerssen of Leiden University for annotating the data. [AKG08] [KA18] [LM14] [NSO+15] [SGN+15] [SHBL16] [SOG+16] [TP10]
Pam Carter, Roger Beech, Domenica Coxon, Martin J. Thomas, and Clare Jinks. Mobilising the experiential knowledge of clinicians, patients and carers for applied health-care research. Contemporary Social Science, 8(3):307{320, 2013.
Payam Karisani and Eugene Agichtein. Did you really just have a heart attack? Proceedings of the 2018 World Wide Web Conference on World Wide Web - WWW 18, 2018.
National Library of Medicine (US). Rxnorm.
Azadeh Nikfarjam, Abeed Sarker, Karen O'Connor, Rachel Ginn, and Graciela Gonzalez. Pharmacovigilance from social media: mining adverse drug reaction mentions using sequence labeling with word embedding cluster features. Journal of the American Medical Informatics Association: JAMIA, 22(3):671{81, 2015.
Abeed Sarker and Graciela Gonzalez. Portable automatic text classi cation for adverse drug reaction detection via multi-corpus training. Journal of Biomedical Informatics, 53:196{207, 2015. Abeed Sarker, Rachel Ginn, Azadeh Nikfarjam, Karen O`Connor, Karen Smith, Swetha Jayaraman, Tejaswi Upadhaya, and Graciela Gonzalez. Utilizing social media data for pharmacovigilance: A review. Journal of Biomedical Informatics, 54:202{212, 2015.
Edin Smailhodzic, Wyanda Hooijsma, Albert Boonstra, and David J. Langley. Social media use in healthcare: A systematic review of e ects on patients and on their relationship with healthcare professionals. BMC Health Services Research, 16(1):442, 2016.
Abeed Sarker, Karen O`Connor, Rachel Ginn, Matthew Scotch, Karen Smith, Dan Malone, and Graciela Gonzalez. Social Media Mining for Toxicovigilance: Automatic Monitoring of Prescription Medication Abuse from Twitter. Drug Safety, 39(3):231{240, 2016.
Suzan Verberne, Anika Batenburg, Remco Sanders, and Mies Van Eenbergen. Social processes of online empowerment on a cancer patient discussion form: using text mining to analyze linguistic patterns of empowerment processes. JMIR Cancer, Forthcoming.