Adverse Drug Event (ADE) extraction from user-generated content has gained popularity as a tool to aid researchers and pharmaceutical companies to monitor side efect of drugs in the wild. Automatic models can rapidly examine large collections of social media texts. However it is currently unknown if such models are robust in face of linguistic phenomena such as negation and speculation, which are pervasive across language varieties. We evaluate three state-of-the-art systems, showing their fragility against negation, and then we introduce two possible strategies to increase the robustness of these models: (i) a pipeline approach, using a specific component for negation detection; (ii) an augmentation of the dataset with artificially negated samples to further train the models. We show that both strategies bring significant increases in performance.
As more users keep reporting their personal experience with drugs on social media, blogs and
health forums, automatic Adverse Drug Event (ADE) detection in social media texts is becoming
a fundamental tool in the field of pharmacovigilance [
Tweet collection
Collection and extraction modules
Classification module
Contains ADE?
No Discard
Yes
Extraction module Which are the ADE? headache arm pain clot formal medical dictionary or ontology.
In the last decade, the Natural Language Processing (NLP) community dedicated a consistent
efort in developing robust methods for mining biomedical information from user-generated
texts, also leading to the creation of several dedicated shared tasks series on ADE detection
(SMM4H – Social Media Mining for Health) [
Detecting the scope of negation and speculation has been object of NLP research for at least
one decade, via both rule-based and machine learning approaches. An early, popular system
was introduced by Chapman et al. [
As of today, the research in biomedical NLP mostly focused on scope detection of negations and speculations per se and on more formal types of texts (e.g. clinical notes, articles). Given the growing demand to process and analyze large collections of user-generated content from social media, we choose to focus on ADE detection on Twitter posts. They are characterized by a noisier and more informal writing style. The goal is to enable to systems to be more successful at distinguishing between factual and non-factual information.
In this paper, introduce an extended dataset to analyze the performance of ADE extraction in presence of asserted and negated Adverse Events. We show that the latest state-of-the-art ADE detection systems cannot recognize and handle negations correctly and introduce two strategies to increase the robustness of existing systems: (i) adding a negation detection module in a pipeline fashion to exclude the negated ADEs predicted by the models; (ii) augmenting the training set with artificially negated samples.
Following the latest advancements in the SMM4H Shared Tasks, we choose three
Transformerbased models that showed high performance on the ADE extraction dataset of SMM4H [
... no headache ...
Collection and extraction modules headache arm pain clot
Negation detection module
Artificial samples Remove intersections
no headache no clot arm pain ... arm pain ... ... no clot ...
... no headache ...
Extraction module
Collection and
extraction modules
arm pain
and are currently at the top of the corresponding leaderboard: BERT [
We analyze two possible strategies to increase the robustness of the baseline models: (i) adding a negation (or speculation) detection module in a pipeline fashion to exclude some incorrect adverse events predicted by the models; (ii) augmenting the training set with artificially created samples. Figure 2 illustrates the two approaches.
We propose a simple pipeline to enhance the robustness of the base models against negation by combining them with a negation detection module. Let us consider a text , a ADE extraction base model ℬ and a negation detection module . Given , ℬ outputs a set of substrings of that are labeled as ADE mentions: ℬ() = {1, . . . , }. Similarly, takes a text and outputs a set of substrings, which are considered to be entities within a negation scope: () = {1, . . . , }.
A combined pipeline model is obtained by discarding all ADE spans ∈ ℬ() that overlap
one of the negation spans ∈ (): ℬ () = { ∈ ℬ() | ∀( ∈ () ∧ ∩ = ∅)}
Modules used We introduce two negation detection modules: NegEx, a Python implementation
[
While there are several datasets for ADE detection on social media texts [
However, most datasets are made of samples that either do or do not contain an ADE (useful to train the Classification module in Figure 1). Because of this, they include a small number of negated ADEs by construction: no particular attention is given to these samples when curating the data and, even when they are present, they are labelled as noADE samples. This makes it harder to study this phenomenon.
We augment the SMM4H19 dataset (the training set for the ADE extraction Task of SMM4H19
[
Recovery of real samples We look for real samples that negate the presence of an ADE using
SMM4H19 and SMM4H20 , the datasets for the binary classification tasks in [
Generation of negated samples We manually create negated versions for the ADE tweets in the test split of SMM4H19. These are meant to be used as additional training samples, to teach the model how to distinguish asserted and negated adverse events. The result of this procedure is a new set of tweets denying the presence of an ADE. As an example, the original tweet “fluoxetine, got me going crazy” was transformed into “fluoxetine, didn’t get me going crazy”.
We split the available data in a train and a test set, both containing the three categories of tweets: ADE, noADE and negADE. Given the small amount of real negADE tweets, we use all of them in the test set to evaluate the performance only on real tweets. Conversely, the training set only contains the manually generated negADE samples.
All the reported results are the average over 5 runs. For the Transformer models we used the
same hyperparameters reported by Portelli et al. [
As a preliminary step for all experiments, the two negation detection models are trained and used to predict the negation scopes for all the test samples once. This allows us to compute the predictions of any pipeline model.
Exp 0 (row 1) To provide a measure of the initial robustness of the base models and their general performance, we train them on the ADE and noADE samples only. The base models have a high number of FP, especially in the negADE category. This strongly suggests that they are not robust against this phenomenon.
Exp 1 (rows 2–3) We test the eficacy of the pipeline negation detection method, applying NegEx and BERTneg to the base models. When combined with NegEx (row 2), the FP decreases by almost 50 points, showing that the regular expression module removes a great number of unwanted predictions. BERTneg decreases the number of FP too, but only by 38 points, being less aggressive than NegEx. However, if we look at P and R in the first three rows, we see that the negation detection modules increase P at the cost of large drops in R: some correct predictions of the base models get discarded (i.e., ADEs that contain a negation such as “After taking this drug I cannot sleep anymore”).
Exp 2 (row 4) We add to the training set all negADE generated samples and train the base models on them to test the efect of augmenting the dataset. This lowers the number of FP predictions for all models as much as using NegEx (compare row 2 and 4), especially on the negADE set. We still observe a drop in R, but less severe than in Exp 1 (less true positives are being discarded). The increase in P is also more noticeable, leading to an overall increase in F1. Exp 3 (rows 5–6) To investigate whether the two methods are complementary in their action, we combine the two strategies, applying the pipeline architecture to the models trained on the augmented dataset. They are in some way complementary, as shown by the further decrease in FP in all categories. However, combining the two approaches might not be the best strategy, as it leads to a further decrease in R.
Observations The results show that introducing a small number of new samples (even if artificial) is the best way to directly increase the model knowledge about the phenomenon. However, this solution could be expensive in absence of annotated data. For this reason, the pipeline models might be a viable alternative, as they maintain the F1 score while still decreasing the number of FP.
In this paper, we evaluate the impact of negations on state-of-the-art ADE detection models. We introduce and compare two strategies to tackle the problem: using a negation detection module and adding negSamp samples in the training set. Both of them bring significant increases in performance. Future work should focus on more refined techniques to accurately model the semantic properties of the samples, also by jointly handling negation and speculation phenomena. This might be an essential requirement for dealing with the noisiness and variety of social media texts.