=Paper=
{{Paper
|id=Vol-3307/paper8
|storemode=property
|title=Detection of Adverse Drug Events from Social Media Texts - Research Project Overview
|pdfUrl=https://ceur-ws.org/Vol-3307/paper8.pdf
|volume=Vol-3307
|authors=Simone Scaboro,Beatrice Portelli,Giuseppe Serra
|dblpUrl=https://dblp.org/rec/conf/aiia/ScaboroP022
}}
==Detection of Adverse Drug Events from Social Media Texts - Research Project Overview ==
https://ceur-ws.org/Vol-3307/paper8.pdf
Detection of Adverse Drug Events from Social Media
Texts – Research Project Overview
Simone Scaboro1 , Beatrice Portelli1 and Giuseppe Serra1
1
Department of Mathematics, Computer Science and Physics, University of Udine, Udine UD 33100, IT
Abstract
This paper presents an overview of the current research project on the detection of Adverse Drug Events
from social media texts led by the Artificial Intelligence Laboratory of Udine (AILAB Udine).
In the latest years, patients started reporting Adverse Drug Events (ADEs) on social media and similar
online outlets, making it necessary to monitor them for pharmacovigilance purposes. For this reason,
systems for the automatic extraction of ADEs are becoming an important research topic in the Natural
Language Processing community.
In this paper we present our research project, focused on the Detection, Extraction and Normalization
of ADEs, detailing its objectives, achievements and future directions.
Keywords
Adverse Drug Events, Social Media, Deep Learning
1. Introduction
Each year, dozens of new drugs and medical compounds are developed, tested and released in
the market. As an example, in 2021, the Food and Drug Administration (FDA) approved the use
of 50 new drugs [1], while the European Medicines Agency (EMA) recommended 92 medicines
for marketing authorization [2]. These regulators are in charge of supervising the process that
leads to the release of these new medicines, authorizing only the products that are deemed safe.
For this reason, new drugs are approved only after rigorous medical trials, which aim to prove
their therapeutic efficacy and their safety. In particular, regulatory authorities are interested
in documenting possible Adverse Drug Events (ADE), that are unexpected reactions or effects
related to the correct use of a medicine.
However, unexpected ADEs might emerge once the new medication is used by a larger
population. This is why pharmaceutical and governance agencies have looked into new ways
to detect these events in large populations, creating Pharmacovigilance programs.
Traditionally, the process to collect ADEs relies on formal reporting methods, based on the
communication between patients, healthcare providers, pharmaceutical companies, and local
PV authorities. ADEs can also be extracted (either manually of automatically) from formal
medical documents, such as Electronic Health Records (EHR) (see [3] for a recent overview).
HC@AIxIA 2022: 1st AIxIA Workshop on Artificial Intelligence For Healthcare, November 30, 2022, Udine, IT
$ scaboro.simone@spes.uniud.it (S. Scaboro); portelli.beatrice@spes.uniud.it (B. Portelli); giuseppe.serra@uniud.it
(G. Serra)
http://conceptbase.sourceforge.net/mjf/ (G. Serra)
0000-0003-2533-1298 (S. Scaboro); 0000-0001-8887-616X (B. Portelli); 0000-0002-4269-4501 (G. Serra)
© 2022 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
CEUR
Workshop
Proceedings
http://ceur-ws.org
ISSN 1613-0073
CEUR Workshop Proceedings (CEUR-WS.org)
Recently, a lot of effort has been put into applying AI models to automatically extract useful
information. In particular, user-generated content such as social media posts present a great
opportunity: an increasing number of patients discuss their health on forums and micro-
blogging platforms, including details on their physical and mental health, as well as feedback
on medications and medical procedures. AI systems could be used to analyze this health-related
chatter on social media, extract useful information, and perform automatic tasks including ADE
detection [4].
However, social media texts introduce a series of new challenges, such as dealing with
highly informal speech and the presence of layman terms, typos, and linguistic phenomena like
humour, irony, speculations and negations that could affect the meaning of the message. These
characteristics make the texts difficult to understand for current natural language understanding
techniques, which often rely on models pretrained on large general-domain corpora based on
Wikipedia, English literature or other resources. These models have an hard time adapting to
the informal language of social media, as they are not used to encounter typos and they also
need to learn new contextual meanings for the words used in metaphors and slang. To tackle
these problems, a growing branch of the Natural Language Processing (NLP) community has
focused on the extraction of ADE mentions from social media texts, encouraging research in
this field through conferences and dedicated challenges. One of the most significant examples
is the Social Media Mining for Health Applications (SMM4H) Shared Task, which is co-located
with top-tier international conferences such as ACL and COLING.
Our international research team led by the Artificial Intelligence Laboratory of Udine (AILAB
Udine)1 is working on an ongoing research project focused on deep learning methods for
automatic handling of ADE mentions in social media texts. In this paper we present a summary
of our activities and the future directions of our work.
2. Current Research Project
Our research project stemmed from the growing importance of social media in the field of
digital pharmacovigilance. The aim of pharmacovigilance systems is to identify Adverse Drug
Events, document them and analyze them. Traditionally, this task has been broken down into
three sub-tasks (see Figure 1):
1. ADE Detection: the binary classification of pieces of text as either containing or not
containing a possible ADE. This initial step is especially needed when working with social
media posts, due to the large volume of input data.
2. ADE Extraction: the extraction of the exact ADE mention(s) from a given texts. During
this step, the model is given a full text and is expected to return only the excerpt which is
relevant to describe the ADE.
3. ADE Normalization: the ADE extracted at the previous step, which is usually written
in informal or colloquial language, is mapped to a formal term belonging to a medical
ontology. This is the last step of process is necessary to identify precisely what kind of
reaction was caused by the drug, and allows medical professionals to perform further
analyses on the aggregated data.
1
http://ailab.uniud.it/
Unilateral
neck swelling original text
MedDRA
prior to
I've got the ADEs terms
starting the
input documents
Nubeqa. Mirena, I've Abdominal
cramping ADE
experienced pain lower
ADE ADE Normali-
I've got the cramping
Detection Extraction zation
Mirena, I've and a little a little bit Genital
experienced bit of of bleeding haemorrhage
cramping bleeding.
and a little
bit of
bleeding.
Figure 1: Example of a pipeline applying the three sub-tasks (ADE Detection, Extraction and Normal-
ization) to two input texts.
Among the three tasks, Extraction and Normalization are the ones on which the community
has focused more in the latest years, as they present multiple challenges connected to the
peculiarities of informal social media language. Additionally, texts mined from different sources,
such as internet forums, tweets, and reddit threads, might have different linguistic characteristics,
which make it harder to build a unique system that works for all of them.
Our research group briefly tackled the task of ADE Detection, to then focus mainly on the
ADE Extraction, its limits and its applications. Finally, we recently tackled the problem of ADE
Normalization, with interesting findings on how to improve the generalization capabilities of
models on large ontologies.
2.1. Materials and Methods
Most of the resources and datasets used in our research are publicly available. The datasets
used in our works are widely used benchmarks for ADE-related tasks on social media texts.
The following is a list of the main resources used in the projects:
• CADEC [5]. Public dataset containing 1,250 posts from the health-related forum “AskaPa-
tient”. Each post is annotated for the presence of ADEs, and each ADE is mapped to the
corresponding medical term in various medical ontologies (MedDRA and SNOMED CT).
• SMM4H’19 [6], SMM4H’20 [7] and SMM4H’22 [8]. Public datasets for the following
shared tasks: SMM4H 2019 - Task 1–3, SMM4H 2020 - Task 3, SMM4H 2022 - Task 1.
They contains 2,000 to 23,000 tweets which mention at least one drug name. A portion of
the tweet contain one or more ADEs, which are also mapped to a term (Preferred Term
or Lowest Level Term) in the MedDRA ontology.
The texts contained in the SMM4H datasets are short, highly informal and rich of typos
and/layman terms. Differently, the forum posts contained in CADEC are longer (up to 5 times
the length of the tweets), more descriptive and sometimes contain more precise medical terms,
as they were posted on an health-related forum. Using datasets with different textual styles
allows us to test and develop models in different scenarios, working towards the creation of
models that can deal with varying levels of informality within texts.
Most of the systems we developed built upon Transformer-based models which use BERT
variants [9]. These large language models proved to be extremely effective in several NLP
tasks, including ADE Detection/Extraction from social media texts. Moreover, over the years
researchers developed several BERT variants pre-trained on medical texts which hold useful
in-domain knowledge for the these tasks (e.g., PubMedBERT [10]).
2.2. ADE Detection
As previously mentioned, our group briefly touched on the topic of ADE Detection, mainly to
test the capabilities of current BERT-based models on the task, and to have an in-house system
ready to use for future tasks.
We developed a simple BERT-based model the binary classification of texts as containing or
not containing an ADE. It consists in a bert-base-uncased model with a binary classification
head. If the text contains more than one sentence, it is split in single sentences and each one is
evaluated for the presence of ADEs. If at least one of the sentences is labeled as positive, the
whole sample is marked as positive for the presence of ADEs.
In Table 1, we compared the performance of this model with the best systems reported in
literature on two datasets: SMM4H’19 and CADEC. The model reaches a good performance on
SMM4H’19, surpassing in Precision the two best models which took part in the shared task.
On CADEC, our system largely outperforms previously reported results in Recall and F1-score.
Since most of the texts in CADEC are composed by several sentences, with an average of 5
sentences per sample, we believe that splitting them and evaluating them separately is what
contributed the most to the increase in performance.
In following works, we empirically replaced the BERT module with other BERT variants
pre-trained on medical texts (e.g., PubMedeBERT) to get small performance boosts in specific
tasks and datasets.
Table 1
Comparison of our BERT-based classifier with the best performing systems on SMM4H’19 and CADEC
for ADE Detection. Performances are reported in terms of Precision (P), Recall (R) and F1-score (F1), the
harmonic mean of P and R.
Dataset Model F1 P R
Chen et al. [11] 64.57 60.79 68.85
SMM4H’19 Ellendorff et al. [12] 60.48 64.78 56.71
Our (BERT-based classifier) 61.93 65.67 58.68
Alimova and Tutubalina [13] 80.3 84.4 77.3
CADEC Alimova and Tutubalina [14] 81.5 – –
Our (BERT-based classifier) 84.3 78.5 90.9
2.3. ADE Extraction
We addressed the task of ADE extraction as a tagging task (or Named Entity Recognition task):
given a text, our goal is to identify whether it contains an ADE and its precise span. To this
end, we applied for the first time SpanBERT [15], a BERT variant specialized in multi-token
text spans, such as those describing ADE. In our experiments [16], we use a SpanBERT model
to generate one embedding for each word in the input text. The embeddings of each word are
then fed to a linear layer to output a 3-class BIO label, using the Begin-Inside-Outside tagging
scheme employed for NER tasks. The B label marks the beginning of an entity, while the I
label is used to mark the following tokens in a multi-token entity. Figure 2 (left) illustrates this
architecture. We also experimented the combination of SpanBERT with a Conditional Random
Field (CRF) [17] module to further de-noise its output. The CRF is applied to the output of the
linear layer, producing another set of BIO labels, as represented in Figure 2 (right).
Input text Token-level labels Input text Token-level labels
(as a sequence of tokens) (as a sequence of tokens)
O O
B B
O BERT- O
BERT-based I I
based CRF
Model I I
B Model B
O O
O O
Figure 2: Architectures used for the ADE Extraction task using BERT-based models and Conditional
Random Fields (CRF). The input of the model is a text, seen as a sequence of tokens. The output is a
series of BIO labels (Begin-Inside-Outside) that mark the presence of ADE entities.
The performance of the system was evaluated on SMM4H’19 and CADEC. The results we
of our experiments show that this approach obtains competitive performance compared to
other models that took part in the SMM4H’19 task, and also other BERT variants. Moreover
SpanBERT+CRF outperforms all the other approaches on both datasets.
The system we developed in currently at the first place on the public leaderboard of the
SMM4H’19 ADE Extraction task.2
2.3.1. Application: A Web Platform to Monitor Adverse Reactions to COVID-19
Vaccines on Twitter.
The development of our ADE Extraction system coincided with the start of the COVID-19
vaccination campaigns at the end of 2020. The diffusion of COVID-19 vaccines through massive
vaccination campaigns brought a lively debate on social media, with thousands of public
discussions about possible ADEs. This was the perfect environment to test the ability of an
automatic system for ADE Extraction, which could also provide insightful information to general
users. In this context, our international research team developed a web platform3 for monitoring
English tweets about COVID-19 vaccines [18].
The objective of the platform is to collect and process tweets, providing visual representations
of the information extracted. This is possible thanks to a set of modules (shown in Figure 3)
that interact with each other. These modules include the ADE Extraction module described
in the previous section. This extracted information is then aggregated and visualized through
2
https://competitions.codalab.org/competitions/20798, Sub-Task 2, Post-evaluation
3
http://ailab.uniud.it/covid-vaccines/
Figure 3: Schema of the modules implemented on the web platform developed to analyze tweets about
COVID-19 vaccines.
an interactive word-cloud that can be filtered by date and vaccine name, permitting the user
to investigate of trends across time and vaccines. In fact, thanks to this tool, is possible to
monitor the tweets that mention a specific drug and obtain an overview of the possible side
effects reported by the users, their frequency, and the original texts from which the ADEs
were extracted. Despite the promising performance of the model on benchmark datasets, it is
important to highlight that the data produced by these current automatic systems still need an
expert’s supervision to be validated, and should not be used without human revision. However,
they are a powerful tool to redirect the experts’ attention to potentially interesting phenomena.
As previously mentioned, the portal is also enriched with other modules that give different
information on the population of users, such as their geographical location, an important piece
of information which could be used to identify patterns in ADE reports. The system also
analyzes the URLs that the users share together with information about the vaccines, which is
important to take into consideration in the fight against misinformation.
The web portal proved to be useful to perform some analyses on social media chatter sur-
rounding the AstraZeneca vaccines, and it is still actively processing data.
2.3.2. Focus: The Effect of Negations and Speculation on ADE Extraction
Analyzing the incorrect outputs of our ADE Extraction system and other baselines systems
we discovered that a great number of False Positives (incorrectly extracted ADEs) were caused
by the presence of particular linguistic phenomena, such as irony, negations and speculations.
These phenomena are pervasive in social media texts, and they seriously hamper the ability of
an automated system to discriminate between factual and nonfactual statements. Therefore, we
studied methods to develop more robust models for ADE Extraction in face of negations and
speculations [19, 20].
In our research, we took into consideration some systems for ADE detection on social media
texts and introduced SNAX, a benchmark dataset to test their performance against negations
and speculations on ADE Extraction from social media texts. The dataset is composed by a set of
samples (tweets) belonging to four categories: tweets containing ADEs, tweets not containing
ADEs, tweets that explicitly negate the presence of an ADE, and tweets that speculate an ADE.
We then introduced two possible strategies to increase the robustness of these models:
• data augmentation: the use of artificially negated and/or speculated samples during the
training of the model;
• negation/speculation detection module: the use of a negation and/or speculation module
as part of the ADE Extraction pipeline;
All the baseline ADE Extraction models that we analyzed are not robust against negations
and speculations. The two strategies that we introduced successfully lower the number of False
Positive predictions of the baseline models. However, they have the drawback of lowering the
Recall of the models, which is especially noticeable when using the second strategy. Combining
both strategies leads to the best results in terms of FP reduction but exacerbates the drop in
Recall, so it is not recommended in pracical scenarios.
With these works, we proved the lack of robustness of the current state-of-the-art models for
the ADE Extraction task, providing a challenging setting to test future models. Furthermore, we
introduced two strategies to address the problem, highlighting the improvements they achieved
and also their weaknesses. In conclusion, we believe that these kinds of studies, which explore
the lack of robustness against different natural language phenomena, should be the backbone
of the analysis of models that deal with informal language.
2.4. ADE Normalization
Currently, our research project is focused on ADE Normalization, that is mapping the ex-
tracted mentions to large formal medical ontologies (e.g., MedDRA, containing over 24K unique
Preferred Terms).
We [21] recently took part in the SMM4H’22 Shared Task for ADE Normalization, using a
system based on GPT-2, a text generation model. It ranked third in the task [8], reaching 76%
normalization accuracy.
However, one of the main challenges of this task is the high cardinality of the output space,
and long tail distribution of the labels in available datasets. This is because current datasets
for this task contain at most 5,000 samples, covering only 200-500 of the possible 24K output
classes, and the label distribution is highly skewed towards few very frequent terms. For this
reason, the models usually perform well on examples that are seen in the training set, but are
unable to generalize on rare or unseen ones.
Therefore, we focused our research on ways to overcome the long tail distribution of the
training terms and increase the generalization capabilities of ADE Normalization models. We
proposed “Ontology Pretraining” (OP) [22], a pre-training strategy based on the hierarchical
nature of the MedDRA medical ontology, that improves the generalization capabilities of large
language models. This pretraining step injects domain knowledge about all the output labels of
the ontology before performing a classical fine-tuning using just the few labels present in the
actual training dataset. The OP strategy uses the Lowest Level Terms (LLT) present in MedDRA
and teaches the ADE Normalization model to map them to MedDRA Preferred Terms (PT), our
actual output labels. LLTs sub-categories of PTs, and are more informal names, which make
them stylistically closer to ADEs. Therefore, a model trained with OP will have an advantage
when presented with ADEs as input.
We performed several experiments on CADEC, SMM4H’20 and a proprietary dataset, using
several ADE Normalization models. The experiments show that using the OP strategy before
classical fine-tuning drastically improves the generalization abilities of all models without dam-
aging their performance on seen concepts. The models trained using OP also showed promising
results for zero-shot cross-dataset normalization, which is interesting in the perspective of
developing a model that can work well across different text typologies.
3. Conclusions and Future Directions
Our research group tackled the tasks of ADE Detection, Extraction and Normalization on social
media texts. The results consisted in the development of effective deep learning models for the
tasks, some of which were also applied to create an online web platform for ADE monitoring.
The activities also highlighted several limitations of current models, which still require further
exploration. For example, the performance of ADE Detection models is still scarce on highly
informal texts, such as tweets. However, these are kind of texts for which the community
needs a robust ADE Detection model, as they could become the input of real-time digital
pharmacovigilance systems. Moreover, ADE Extraction models still lack the ability to deal with
some pervasive linguistic phenomena (e.g., irony and humor), and the proposed solutions of
negation/speculation detection could be improved to reduce drops in recall. Right now, ADE
Extraction models cannot be blindly applied to real-time streams of social media posts, as
they lack to ability to distinguish personal ADE reports from news pieces and second-hand
reports (i.e., recounting what happened to another person). Even more importantly, they
are not able to distinguish real ADE reports from maliciously-constructed posts that spread
misinformation. Therefore, there is a need to look into ways to carefully integrate them in online
pharmacovigilance activities, with the aid of other systems and expert human supervision.
As regards the future directions of this project, one of the top priorities is extending the
current systems to other languages. SMM4H recently introduced small datasets in Spanish,
French and Russian, but there is still a lack in resources and models for languages other than
English. Medical ontologies, on the other hand, come with several official translations, so it
would be interesting to explore the idea of leveraging them to solve other tasks. We also plan to
focus more on term normalization, incorporating different ontologies (e.g., SNOMED CT and
UMLS). This will help in two ways: by adding new complementary knowledge to train the AI
systems, and by creating more bridges between the outputs of the NLP systems and the formal
medical world. Finally, it would be interesting to move towards to extraction and normalization
of other medical entities in informal texts, such as drugs and diseases.
References
[1] B. G. de la Torre, F. Albericio, The pharmaceutical industry in 2021. an analysis of fda
drug approvals from the perspective of molecules, Molecules 27 (2022). doi:10.3390/
molecules27031075.
[2] European Medicines Agency, Human medicines highlights 2021, https://www.ema.europa.
eu/documents/report/human-medicines-highlights-2021_en.pdf, 2022. Accessed: 2022-10-
07.
[3] C. Feng, D. Le, A. B. McCoy, Using Electronic Health Records to Identify Adverse Drug
Events in Ambulatory Care: A Systematic Review, Applied Clinical Informatics 10 (2019)
123–128.
[4] H. Yang, C. C. Yang, Using health-consumer-contributed data to detect adverse drug
reactions by association mining with temporal analysis, ACM Trans. Intell. Syst. Technol.
6 (2015). URL: https://doi.org/10.1145/2700482. doi:10.1145/2700482.
[5] S. Karimi, A. Metke-Jimenez, M. Kemp, C. Wang, Cadec: A Corpus of Adverse Drug Event
Annotations, J. Biomed. Inform. 55 (2015) 73–81.
[6] D. Weissenbacher, A. Sarker, A. Magge, A. Daughton, K. O’Connor, M. Paul, G. Gonzalez,
Overview of the Social Media Mining for Health (SMM4H) Shared Tasks at ACL 2019, in:
Proceedings of the ACL Workshop on Social Media Mining for Health Applications, 2019.
[7] G. Gonzalez-Hernandez, A. Z. Klein, I. Flores, D. Weissenbacher, A. Magge, K. O’Connor,
A. Sarker, A.-L. Minard, E. Tutubalina, Z. Miftahutdinov, I. Alimova, Proceedings of the
COLING Social Media Mining for Health Applications Workshop & Shared Task, 2020.
URL: https://aclanthology.org/2020.smm4h-1.0.
[8] D. Weissenbacher, J. Banda, V. Davydova, D. Estrada Zavala, L. Gasco Sánchez, Y. Ge,
Y. Guo, A. Klein, M. Krallinger, M. Leddin, A. Magge, R. Rodriguez-Esteban, A. Sarker,
L. Schmidt, E. Tutubalina, G. Gonzalez-Hernandez, Overview of the seventh social media
mining for health applications (#SMM4H) shared tasks at COLING 2022, in: Proceedings
of The Seventh Workshop on Social Media Mining for Health Applications, Workshop
& Shared Task, Association for Computational Linguistics, Gyeongju, Republic of Korea,
2022, pp. 221–241. URL: https://aclanthology.org/2022.smm4h-1.54.
[9] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of Deep Bidirectional
Transformers for Language Understanding, in: Proceedings of NAACL, 2019.
[10] Y. Gu, R. Tinn, H. Cheng, M. Lucas, N. Usuyama, X. Liu, T. Naumann, J. Gao, H. Poon,
Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing,
arXiv preprint arXiv:2007.15779 (2020).
[11] S. Chen, Y. Huang, X. Huang, H. Qin, J. Yan, B. Tang, HITSZ-ICRC: A report for SMM4H
shared task 2019-automatic classification and extraction of adverse effect mentions in
tweets, in: Proceedings of the Fourth Social Media Mining for Health Applications
(#SMM4H) Workshop & Shared Task, Association for Computational Linguistics, Florence,
Italy, 2019, pp. 47–51. doi:10.18653/v1/W19-3206.
[12] T. Ellendorff, L. Furrer, N. Colic, N. Aepli, F. Rinaldi, Approaching SMM4H with merged
models and multi-task learning, in: Proceedings of the Fourth Social Media Mining for
Health Applications (#SMM4H) Workshop & Shared Task, Association for Computational
Linguistics, Florence, Italy, 2019, pp. 58–61. doi:10.18653/v1/W19-3208.
[13] I. Alimova, E. Tutubalina, Automated detection of adverse drug reactions from social
media posts with machine learning, in: W. M. van der Aalst, D. I. Ignatov, M. Khachay,
S. O. Kuznetsov, V. Lempitsky, I. A. Lomazova, N. Loukachevitch, A. Napoli, A. Panchenko,
P. M. Pardalos, A. V. Savchenko, S. Wasserman (Eds.), Analysis of Images, Social Networks
and Texts, Springer International Publishing, Cham, 2018, pp. 3–15.
[14] I. Alimova, E. Tutubalina, Detecting adverse drug reactions from biomedical texts with
neural networks, in: Proceedings of the 57th Annual Meeting of the Association for
Computational Linguistics: Student Research Workshop, Association for Computational
Linguistics, Florence, Italy, 2019, pp. 415–421. doi:10.18653/v1/P19-2058.
[15] M. Joshi, D. Chen, Y. Liu, D. S. Weld, L. Zettlemoyer, O. Levy, SpanBERT: Improving
Pre-training by Representing and Predicting Spans, Transactions of the Association for
Computational Linguistics 8 (2020) 64–77.
[16] B. Portelli, D. Passabì, E. Lenzi, G. Serra, E. Santus, E. Chersoni, Improving Adverse Drug
Event Extraction with SpanBERT on Different Text Typologies, Springer International
Publishing, Cham, 2022, pp. 87–99. doi:10.1007/978-3-030-93080-6\_8.
[17] J. Lafferty, A. Mccallum, F. Pereira, Conditional Random Fields: Probabilistic Models for
Segmenting and Labeling Sequence Data, in: Proc. of ICML 2001, Morgan Kaufmann
Publishers Inc., San Francisco, CA, USA, 2001, pp. 282—-289.
[18] B. Portelli, S. Scaboro, R. Tonino, E. Chersoni, E. Santus, G. Serra, Monitoring user opinions
and side effects on covid-19 vaccines in the twittersphere: Infodemiology study of tweets,
J Med Internet Res 24 (2022) e35115. doi:10.2196/35115.
[19] S. Scaboro, B. Portelli, E. Chersoni, E. Santus, G. Serra, NADE: A benchmark for robust
adverse drug events extraction in face of negations, in: Proceedings of the Seventh
Workshop on Noisy User-generated Text (W-NUT 2021), Association for Computational
Linguistics, Online, 2021, pp. 230–237. doi:10.18653/v1/2021.wnut-1.26.
[20] S. Scaboro, B. Portelli, E. Chersoni, E. Santus, G. Serra, Increasing adverse drug events
extraction robustness on social media: case study on negation and speculation, 2022.
doi:10.48550/ARXIV.2209.02812.
[21] B. Portelli, S. Scaboro, E. Chersoni, E. Santus, G. Serra, AILAB-Udine@SMM4H’22: Limits
of transformers and BERT ensembles, in: Proceedings of The Seventh Workshop on
Social Media Mining for Health Applications, Workshop & Shared Task, Association
for Computational Linguistics, Gyeongju, Republic of Korea, 2022, pp. 130–134. URL:
https://aclanthology.org/2022.smm4h-1.36.
[22] B. Portelli, S. Scaboro, E. Santus, H. Sedghamiz, E. Chersoni, G. Serra, Generalizing over
Long Tail Concepts for Medical Term Normalization, in: Proceedings of The 2022 Confer-
ence on Empirical Methods in Natural Language Processing, Association for Computational
Linguistics, Abu Dhabi, United Arab Emirates, 2022.