=Paper=
{{Paper
|id=Vol-3740/paper-282
|storemode=property
|title=Conspiracy vs Critical Thinking Using an Ensemble of Transformers with Data Augmentation
Techniques
|pdfUrl=https://ceur-ws.org/Vol-3740/paper-282.pdf
|volume=Vol-3740
|authors=Angelo Maximilian Tulbure,Mariona Coll Ardanuy
|dblpUrl=https://dblp.org/rec/conf/clef/TulbureA24
}}
==Conspiracy vs Critical Thinking Using an Ensemble of Transformers with Data Augmentation
Techniques==
https://ceur-ws.org/Vol-3740/paper-282.pdf
Conspiracy vs Critical Thinking Using an Ensemble of
Transformers with Data Augmentation Techniques
Notebook for PAN at CLEF 2024
Angelo Maximilian Tulbure1,2 , Mariona Coll Ardanuy3
1
Universitat Politècnica de València, València, Spain
2
Politecnico di Milano, Milan, Italy
3
Pattern Recognition and Human Language Technology Research Center, Universitat Politècnica de València, València, Spain
Abstract
This paper provides an overview of our contributions to the PAN at CLEF2024 Oppositional thinking analysis shared
task, which focuses on distinguishing between conspiratorial and critical thinking narratives. The competition
featured two main tasks. The first task is a binary classification task that aims at determining whether a text is
conspiratorial or critical. The second task is a span-level detection task, in which the goal is to detect elements of
oppositional narratives in the texts. Two annotated datasets, one in English and one in Spanish, were provided,
each of 5K telegram comments. Our best-performing approaches combined custom fine-tuned Transformer
models with data augmentation techniques. We achieved an F1-Score of 0.8917 for English and of 0.8293 for
Spanish for task 1, and a span-F1 score of 0.6279 for English and 0.6129 for Spanish for task 2. Our task 2 approach
achieved the best results in the shared task for both English and Spanish.
Keywords
PAN 2024, oppositional thinking, critical thinking, conspiracy theories, binary classification, token classification,
ensembling models, data augmentation
1. Introduction
Conspiracy theories offer elaborate explanations for significant events, attributing them to hidden
schemes by secretive and powerful groups. Recently, there has been increasing interest in automatically
detecting these theories in text, often framed as a binary classification problem, with some approaches
exploring multi-label or multi-class classification. However, key issue with existing methods is their
difficulty in distinguishing between critical thinking and conspiracy theories [1]. This distinction is
crucial, because misclassifying critical perspectives as conspiracies can inadvertently lead individuals to
engage more deeply with conspiracy communities. As argued in Korenčić et al. [1], conspiracy theories
often proliferate rapidly on social media, leading to widespread misinformation and potential harm. In
contrast, critical thinking is essential for informed decision-making and healthy public discourse.
The shared task “Oppositional thinking analysis: Conspiracy theories vs critical thinking narratives”1
[1], which is part of PAN at CLEF2024 [2, 3] deals with the problem of distinguishing between conspiracy
theories and critical thinking. The task focuses on two primary objectives: the binary classification
of texts as either conspiratorial or critical, and the span-level detection of specific elements within
these oppositional narratives. The datasets provided for these tasks include annotated texts from
English and Spanish sources, each consisting of 5,000 telegram comments. These datasets serve as a
comprehensive resource for developing and evaluating models capable of handling multilingual data
and diverse narrative structures.
Our approach to these tasks involved the use of custom fine-tuned Transformer models, which were
enhanced through data augmentation techniques. For the binary classification task, we employed a Soft
Voting Ensembling method combining multiple Transformer models to improve robustness and accuracy.
CLEF 2024: Conference and Labs of the Evaluation Forum, September 09–12, 2024, Grenoble, France
$ angelomaximilian.tulbure@mail.polimi.it (A. M. Tulbure); mcoll@prhlt.upv.es (M. Coll Ardanuy)
0000-0001-8455-7196 (M. Coll Ardanuy)
© 2024 Copyright for this paper by its authors. Use permitted under Creative Commons License Attribution 4.0 International (CC BY 4.0).
1
https://pan.webis.de/clef24/pan24-web/oppositional-thinking-analysis.html.
CEUR
ceur-ws.org
Workshop ISSN 1613-0073
Proceedings
For the span-level detection task, we treated the problem as a token classification task, segmenting text
into sentences to mitigate issues related to text length limitations in Transformer models.
2. Systems Overview
In this section, we describe our submitted systems. Our approaches for distinguishing between critical
and conspiracy texts (Task 1) are described in Section 2.1, and our approaches for detecting elements of
oppositional narratives (Task 2) in Section 2.2.
2.1. Task 1: Distinguishing between Critical and Conspiracy Texts
For Task 1, the general approach involved fine-tuning Transformer-based models and applying data
augmentation techniques. Both English and Spanish datasets were processed similarly. The main
difference between Run 1 and Run 2 lies in the method used to make predictions. In Run 1, only the
best model checkpoint was used to make predictions. In Run 2, an ensembling method was employed,
which combined the predictions from multiple models.
The approach for both languages involved several key steps. We experimented with various
Transformer-based models for fine-tuning and different hyperparameters. In addition to model selection
and training, data augmentation played an important role in our approach. To increase the diversity and
quantity of training data, we applied translation-based augmentation. For this, for English, the Spanish
dataset was translated into English using the Helsinki-NLP/opus-mt-es-en2 model [4]; and,
for Spanish, the English dataset was translated into Spanish using the Helsinki-NLP/opus-mt-
en-es3 model [4]. This method helped in creating a more varied training set, enabling the models to
generalize better and perform more effectively on unseen data.
In Run 1, predictions were made using the best model checkpoints identified during the training
phase. However, in Run 2, we enhanced the prediction process by employing an ensembling method.
This is described in more detail in the following two subsections.
2.1.1. Run 1 task 1
In Run 1, we focused on using the best model checkpoint to make predictions for both English and
Spanish datasets. For English, the facebook/roberta-base4 [5] model was used. This model is
known for its robust performance on various NLP tasks. For Spanish, the dccuchile/bert-base-
spanish-wwm-uncased5 [6] model was used.
2.1.2. Run 2 task 1
In Run 2, an ensembling approach was used. The general approach involved using a Soft Voting Ensem-
bling method composed of three custom fine-tuned Transformer-based models and data augmentation.
We used the following three models for English:
1. facebook/roberta-base6 [5]
2. google/bert-base-uncased7 [7]
3. allenai/scibert_scivocab_uncased8 [8]
For Spanish, we used the following three models:
2
https://huggingface.co/Helsinki-NLP/opus-mt-es-en
3
https://huggingface.co/Helsinki-NLP/opus-mt-en-es
4
https://huggingface.co/FacebookAI/roberta-base
5
https://huggingface.co/dccuchile/bert-base-spanish-wwm-uncased
6
https://huggingface.co/FacebookAI/roberta-base
7
https://huggingface.co/google-bert/bert-base-uncased
8
https://huggingface.co/allenai/scibert_scivocab_uncased
2
1. dccuchile/bert-base-spanish-wwm-uncased9 [6]
2. PlanTL-GOB-ES/bsc-bio-ehr-es-pharmaconer10 [9]
3. bertin-project/bertin-roberta-base-spanish11 [10]
For ensembling, the best checkpoint from each fine-tuned Spanish model was used in a Soft Voting
ensemble for predictions. This approach involved combining predictions from multiple models using a
Soft Voting ensemble [11], which significantly improved the overall accuracy and robustness of the
system. The Soft Voting process involves averaging the predicted probabilities of each category from
the different models and then making the final prediction based on the highest average probability.
By integrating the strengths of different models, the ensembling method provided a more reliable
and precise set of predictions, ensuring higher performance in distinguishing between critical and
conspiracy texts.
2.2. Task 2: Detecting Elements of Oppositional Narratives
We approached Task 2 by fine-tuning a transformer model with a token classification head, therefore
treating it as a token classification problem. Having only one head (instead of a classification head per
label, as is implemented in the provided baseline [1]) precluded the possibility of overlapping spans,
but offered increased simplicity and reduced computational expense instead. While the provided data
was annotated at the document-level, we transformed it so that we could train the token classifier at
the sentence-level instead. Segmenting the text into sentences overcame the problem of transformers
truncating texts that are longer than the maximum length size, ensuring no data was lost during training
or testing. The main difference between Run 1 and Run 2 was that, in Run 1, the best model checkpoint
was used without additional training, while in Run 2, the best model checkpoint was retrained for one
more epoch using the entire dataset as training.
We also performed data augmentation. We could not easily augment the data through translation,
because working at the span-level means that the annotated spans are provided in terms of indices
that match the original text. Therefore, we used an alternative data augmentation technique which
consisted in replacing words in the texts by synonyms or semantically-related words, using static
word embeddings (word2vec) [12] in combination with SpaCy [13]. This process ensured that the total
number of words remained the same to maintain consistency with the start and end spacy tokens.
By introducing synonym replacements, we created a more varied dataset, which helped the models
generalize better and perform more effectively on unseen data. In English, we used the GoogleNews
-vectors-negative30012 static word embeddings while, in Spanish, we used the FastText
embeddings13 from the Spanish Unannotated Corpora.
2.2.1. Run 1 task 2
In Run 1, we used the best model checkpoint without additional training. For Task 2 in English, the
facebook/roberta-base14 [5] model was employed. For the Spanish dataset, the PlanTL-GOB
-ES/roberta-base-bne15 [14] model was used.
2.2.2. Run 2 task 2
In Run 2, the best model checkpoint from Run 1 was trained for one more epoch using the augmented
dataset. This final model checkpoint was then used to detect the elements of oppositional narratives in
the test dataset.
9
https://huggingface.co/dccuchile/bert-base-spanish-wwm-uncased
10
https://huggingface.co/PlanTL-GOB-ES/bsc-bio-ehr-es-pharmaconer
11
https://huggingface.co/bertin-project/bertin-roberta-base-spanish
12
https://github.com/mmihaltz/word2vec-GoogleNews-vectors
13
https://github.com/dccuchile/spanish-word-embeddings
14
https://huggingface.co/FacebookAI/roberta-base
15
https://huggingface.co/PlanTL-GOB-ES/roberta-base-bne
3
2.3. Experiments Setup
Experiments were conducted on an NVIDIA GeForce GTX 1080 (8 GB memory). To ensure uniformity
and comparability of results, the same experimental setting was consistently applied across all tasks,
runs, and languages under study. The experimental setup was meticulously designed to ensure optimal
performance and efficient resource utilization.
StratifiedKFold cross-validation [15] with 3 folds was used to ensure robust performance across
different subsets of the data. This method involves splitting the dataset into 𝑘 folds while preserving the
percentage of samples for each class. For each run, models were trained on 𝑘 − 1 folds and validated on
the remaining fold, rotating this process 𝑘 times to ensure every data point was used for both training
and validation.
The training process spanned 15 epochs. A weight decay of 0.01 was applied as a regularization
technique to penalize large weights. A custom linear learning rate scheduler was employed, adjusting
the learning rate from an initial value of 2e-5 to a final value of 2e-6 over the total number of training
epochs. Gradient accumulation steps were set to 4, effectively increasing the batch size without inflating
the memory footprint by accumulating gradients over multiple steps before updating the model’s
weights. The training batch size per device was dynamically set based on available GPU memory,
managed by a custom callback designed to dynamically adjust the batch size used during training and
evaluation based on the available GPU memory. This ensures efficient resource utilization and prevents
memory-related issues during training, especially when dealing with varying data sizes and model
complexities. Upon completion of training, the best model, as determined by the F1 score, was loaded.
3. Dataset
Participants were provided with a JSON file containing all texts in the training dataset along with
their annotations. Each text is represented by a dictionary that includes the ID, tokenized text, binary
category, and span annotations. Span annotations are provided as a list of dictionaries, with each
dictionary representing an annotated span and detailing the span’s category and text, specified by the
start and end characters. The training subset, comprising 4000 records, was released with all annotations,
while the test subset, consisting of 1000 records, was released only with “id” and the “text” field [1, 2].
(a) binary categories for the English dataset (b) binary categories for the Spanish dataset
Figure 1: Distribution of binary categories for the English and Spanish datasets.
4
Table 1
Distribution of binary labels for the English and Spanish datasets.
Label English (%) Spanish (%)
CRITICAL 2621 65.53% 2538 63.45%
CONSPIRACY 1379 34.48% 1462 36.55%
As summarized in Table 1 and shown in Figures 1a and 1b, the binary classification task for distinguishing
critical from conspiracy texts reveals an inherent class imbalance in both the English and Spanish
datasets. The English dataset has 65.53% critical texts compared to 34.48% conspiracy texts, while the
Spanish dataset shows a similar trend with 63.45% critical and 36.55% conspiracy texts. This imbalance
poses a challenge for model training. The similarity in the proportions of critical and conspiracy texts
across both languages suggested that the models trained on one dataset might be adaptable to the other
with minimal adjustments. That was indeed the approach adopted for the shared task.
(a) Distribution of span text categories for the English dataset.
(b) Distribution of span text categories for the Spanish dataset.
Figure 2: Distribution of span text categories for the English and Spanish datasets. The label "x" represents the
texts where no label appears for the task.
5
Table 2
Distribution of span text labels for the English and Spanish datasets.
Label English (%) Spanish (%)
CAMPAIGNER 5096 22.70% 3285 17.63%
AGENT 5082 22.63% 2698 14.47%
NEGATIVE_EFFECT 4387 19.54% 5770 30.96%
VICTIM 3517 15.67% 4213 22.61%
FACILITATOR 2763 12.31% 2174 11.67%
OBJECTIVE 1602 7.14% 493 2.65%
The token classification task aims to identify key elements within oppositional narratives. As shown in
Figures 2a and 2b and in Table 2, the distribution of these elements varies significantly between the
English and Spanish datasets.
4. Results
The official evaluation metrics were different for each subtask. For subtask 1, which involved distinguish-
ing between critical and conspiracy narratives, the Matthews Correlation Coefficient (MCC) was used
as the primary evaluation metric [16]. Additionally, the F1-macro, F1-conspiracy and F1-critical metrics
were provided for subtask 1. For subtask 2, focusing on span-level detection of narrative elements,
the macro-averaged span-F1 was used as the primary evaluation metric [17]. In addition, the span-P,
span-R and micro-span-F1 metrics were reported for subtask 2.
The shared task organizers provided two baseline models for these tasks. For subtask 1, the baseline
is a standard BERT classifier [7]. For subtask 2, the baseline is a BERT-based multi-task token classifier
with separate classification heads and a common transformer backbone [18]. The baselines utilize either
English or Spanish BERT models, depending on the language. The performances of our approaches and
the baselines are reported in Table 3 (Subtask 1) and in Table 4 (Subtask 2).
Table 3
Performance metrics for Subtask 1.
Subtask 1
MCC F1-macro F1-conspiracy F1-critical
English_baseline 0.7964 0.8975 0.8632 0.9318
English_run1 0.7574 0.8769 0.8338 0.9200
English_run2 0.7872 0.8917 0.8536 0.9297
Spanish_baseline 0.6681 0.8339 0.7872 0.8806
Spanish_run1 0.6147 0.7950 0.7179 0.8720
Spanish_run2 0.6722 0.8293 0.7699 0.8887
Table 4
Performance metrics for Subtask 2.
Subtask 2
span-F1 span-P span-R micro-span-F1
English_baseline 0.5323 0.4684 0.6334 0.4998
English_run1 0.6293 0.5832 0.6856 0.6074
English_run2 0.6279 0.5859 0.6790 0.6120
Spanish_baseline 0.4934 0.4533 0.5621 0.4952
Spanish_run1 0.6089 0.5997 0.6193 0.6051
Spanish_run2 0.6129 0.6159 0.6129 0.6108
6
5. Discussion
Tables 3 and 4 offer a summary of how the models performed in distinguishing between conspiratorial
and critical thinking narratives and in detecting narrative elements within texts. In this section, we
discuss the reported results.
5.1. Subtask 1: Binary Classification of Conspiratorial vs. Critical Thinking
5.1.1. English Results task 1
In our analysis of the English dataset, the first run exhibited good performance metrics. The Matthews
Correlation Coefficient (MCC) was 0.7574, indicating a robust ability to distinguish between different
narrative types. The F1-macro score of 0.8769 further supported the model’s high overall classification
capability. Notably, the F1 score for critical thinking texts was 0.92, compared to 0.8338 for conspiracy
texts. This disparity suggests that the model more effectively identified critical thinking. However,
it is important to note that these results are below the baseline, which had an MCC of 0.7964 and an
F1-macro score of 0.8975. This baseline is indeed a hard baseline and difficult to beat.
In the second run, the ensembled model demonstrated improved reliability, with the MCC rising to
0.7872. The F1-macro score also increased to 0.8917, indicating enhanced overall performance. The F1
scores for conspiracy and critical texts were 0.8536 and 0.9297, respectively, showing more balanced
and accurate classifications. Despite these improvements, the model still did not surpass the baseline,
highlighting the baseline’s strong performance and the challenges in achieving higher accuracy.
5.1.2. Spanish Results task 1
For the Spanish dataset, the first run showed moderate performance with an MCC of 0.6147, indicating
a need for further improvement. The F1-macro score was 0.795, reflecting decent overall performance
but highlighting areas for enhancement. The model struggled more with conspiratorial texts, achieving
an F1 score of 0.7179 for conspiracy versus 0.872 for critical texts, likely due to the specific linguistic
challenges presented by the Spanish language. The baseline for Spanish had an MCC of 0.6681 and an
F1-macro score of 0.8339, indicating that the baseline was also strong for this language.
In the second run, there was a noticeable improvement in performance. The MCC increased to 0.6722,
and the F1-macro score rose to 0.8293, indicating better overall performance. The F1 scores for conspiracy
and critical texts improved significantly to 0.7699 and 0.8887, respectively. These improvements suggest
that the ensembled model became more adept at handling linguistic features specific to Spanish, leading
to more balanced and accurate classifications. This second run beat the strong baseline performance,
reflecting significant progress.
5.2. Subtask 2: Span-level Detection of Narrative Elements
5.2.1. English Results task 2
In the first run for the English dataset, the model achieved a span-P of 0.5832 and a span-R of 0.6856,
indicating moderate precision but better recall. The span-F1 score was 0.6293, and the micro-span-F1
score was 0.6074, suggesting a balanced performance with a need for improvement in precision. The
baseline for this task had a span-F1 score of 0.5323 and a micro-span-F1 of 0.4998, showing that our
model performed significantly better than the strong baseline.
In the second run, there was a slight improvement in precision, with a span-P of 0.5859 and a span-R
of 0.679. The span-F1 score remained relatively consistent at 0.6279, and the micro-span-F1 score
increased marginally to 0.6120. These modest enhancements reflect steady progress in performance
and show that our model maintained competitive performance with the baseline.
7
5.2.2. Spanish Results task 2
For the Spanish dataset, the first run had a span-P of 0.5997 and a span-R of 0.6193. The span-F1 score
was 0.6089, and the micro-span-F1 score was 0.6051. The model performed significantly better at span
detection in Spanish compared to English, possibly due to distinct narrative markers in the language.
The baseline for this task had a span-F1 score of 0.4934 and a micro-span-F1 of 0.4952, indicating that
our model significantly outperformed the baseline in both metrics. In the second run, the performance
improved, with a span-P of 0.6159, a span-R of 0.6129, and a span-F1 of 0.6129, maintaining a significant
advantage over the baseline.
Overall, retraining the model on the entire dataset for one epoch, using the augmented dataset
without a validation and test set, resulted in improved performance during the second run.
5.3. Error analysis
In the future, we will thoroughly investigate why we achieved the best results in the competition in
the second, more challenging task, but not in the first, theoretically easier one. By examining the
confusion matrices for binary classification, we can pinpoint where the model’s predictions deviate
from actual categories. For instance, in the Spanish dataset, the model misclassified "CRITICAL" texts as
"CONSPIRACY" with a 5.52% error rate and "CONSPIRACY" texts as "CRITICAL" with a 31.4% error rate.
In the English dataset, these error rates were 4.12% and 19.7%, respectively, indicating greater difficulty
in detecting "CONSPIRACY" texts, especially in the Spanish dataset. Misclassifications may arise from
language ambiguity, training data limitations, and cultural nuances. For token classification, both
datasets exhibited significant issues with spans that should not be annotated, often being misclassified
into various categories. This could be due to model overconfidence, lack of enough training data, or to
the subjective nature of the task.
6. Conclusion and Future work
The detailed analysis of the results highlights the significant progress made in distinguishing between
conspiratorial and critical thinking narratives and detecting narrative elements within texts. The
continuous improvements observed between runs emphasize the importance of data augmentation,
model fine-tuning, and language-specific adaptations.
Future work should focus on refining these models further, exploring advanced augmentation tech-
niques, and incorporating diverse datasets to improve generalizability. Additionally, developing spe-
cialized tools for intergroup conflict analysis and enhancing content moderation strategies will be
crucial for addressing the challenges posed by misinformation and fostering a healthier information
environment.
Acknowledgements
The work of Mariona Coll Ardanuy is funded by the research project FairTransNLP, grant PID2021-
124361OB-C31, funded by MCIN/AEI/10.13039/501100011033 and by ERDF, EU A way of making
Europe.
References
[1] D. Korenčić, B. Chulvi, X. Bonet-Casals, M. Taulé, P. Rosso, F. Rangel, Overview of the oppositional
thinking analysis PAN task at CLEF 2024, in: G. Faggioli, N. Ferro, P. Galuvakova, A. García Seco
de Herrera (Eds.), Working Notes of CLEF 2024 – Conference and Labs of the Evaluation Forum,
2024.
8
[2] J. Bevendorff, X. Bonet-Casals, B. Chulvi, D. Dementieva, A. Elnagar, D. Freitag, M. Fröbe, D. Ko-
renčić, M. Mayerl, A. Mukherjee, A. Panchenko, M. Potthast, F. Rangel, P. Rosso, A. Smirnova,
E. Stamatatos, B. Stein, M. Taulé, D. Ustalov, M. Wiegmann, E. Zangerle, Overview of PAN
2024: Multi-author writing style analysis, multilingual text detoxification, oppositional thinking
analysis, and generative AI authorship verification, in: Experimental IR Meets Multilinguality,
Multimodality, and Interaction. Proceedings of the Fourteenth International Conference of the
CLEF Association (CLEF 2024), Lecture Notes in Computer Science, Springer, Berlin Heidelberg
New York, 2024.
[3] A. A. Ayele, N. Babakov, J. Bevendorff, X. Bonet-Casals, B. Chulvi, D. Dementieva, A. Elnagar,
D. Freitag, M. Fröbe, D. Korenčić, M. Mayerl, D. Moskovskiy, A. Mukherjee, A. Panchenko, M. Pot-
thast, F. Rangel, N. Rizwan, P. Rosso, F. Schneider, A. Smirnova, E. Stamatatos, B. Stein, M. Taulé,
D. Ustalov, X. Wang, M. Wiegmann, S. M. Yimam, E. Zangerle, Overview of PAN 2024: Multi-
author writing style analysis, multilingual text detoxification, oppositional thinking analysis, and
generative AI authorship verification - condensed lab overview, in: Experimental IR Meets Multi-
linguality, Multimodality, and Interaction. Proceedings of the Fifteenth International Conference
of the CLEF Association CLEF-2024, 2024.
[4] J. Tiedemann, S. Thottingal, OPUS-MT — Building open translation services for the World, in:
Proceedings of the 22nd Annual Conferenec of the European Association for Machine Translation
(EAMT), Lisbon, Portugal, 2020.
[5] Y. Liu, M. Ott, N. Goyal, J. Du, M. Joshi, D. Chen, O. Levy, M. Lewis, L. Zettlemoyer, V. Stoy-
anov, Roberta: A robustly optimized BERT pretraining approach, CoRR abs/1907.11692 (2019).
arXiv:1907.11692.
[6] J. Cañete, G. Chaperon, R. Fuentes, J.-H. Ho, H. Kang, J. Pérez, Spanish pre-trained BERT model
and evaluation data, in: Practical ML for Developing Countries Workshop at the International
Conference on Learning Representations (ICLR 2020), 2020.
[7] J. Devlin, M.-W. Chang, K. Lee, K. Toutanova, BERT: Pre-training of deep bidirectional transformers
for language understanding, in: Proceedings of the 2019 Conference of the North American Chapter
of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long
and Short Papers), Association for Computational Linguistics, Minneapolis, Minnesota, 2019, pp.
4171–4186. URL: https://www.aclweb.org/anthology/N19-1423. doi:10.18653/v1/N19-1423.
[8] I. Beltagy, K. Lo, A. Cohan, SciBERT: A pretrained language model for scientific text, in: K. Inui,
J. Jiang, V. Ng, X. Wan (Eds.), Proceedings of the 2019 Conference on Empirical Methods in
Natural Language Processing and the 9th International Joint Conference on Natural Language
Processing (EMNLP-IJCNLP), Association for Computational Linguistics, Hong Kong, China, 2019,
pp. 3615–3620. doi:10.18653/v1/D19-1371.
[9] C. P. Carrino, J. Llop, M. Pàmies, A. Gutiérrez-Fandiño, J. Armengol-Estapé, J. Silveira-Ocampo,
A. Valencia, A. Gonzalez-Agirre, M. Villegas, Pretrained biomedical language models for clinical
nlp in spanish, in: Proceedings of the 21st Workshop on Biomedical Language Processing, 2022,
pp. 193–199.
[10] J. de la Rosa, E. G. Ponferrada, M. Romero, P. Villegas, P. González de Prado Salas, M. Grandury,
Bertin: Efficient pre-training of a spanish language model using perplexity sampling, Proce-
samiento del Lenguaje Natural 68 (2022) 13–23.
[11] A. Manconi, G. Armano, M. Gnocchi, L. Milanesi, A soft-voting ensemble classifier for detecting
patients affected by covid-19, Applied Sciences 12 (2022) 7554. doi:10.3390/app12157554.
[12] T. Mikolov, I. Sutskever, K. Chen, G. S. Corrado, J. Dean, Distributed representations of words and
phrases and their compositionality, Advances in Neural Information Processing Systems 26 (2013).
[13] M. Honnibal, I. Montani, S. Van Landeghem, A. Boyd, spaCy: Industrial-strength natural language
processing in python, 2020. URL: 10.5281/zenodo.1212303.
[14] A. Gutiérrez-Fandiño, J. Armengol-Estapé, M. Pàmies, J. Llop-Palao, J. Silveira-Ocampo, C. P. Car-
rino, C. Armentano-Oller, C. Rodriguez-Penagos, A. Gonzalez-Agirre, M. Villegas, MarIA: Spanish
language models, Procesamiento del Lenguaje Natural 68 (2022). doi:10.26342/2022-68-3.
[15] T. Fontanari, T. C. Fróes, M. Recamonde-Mendoza, Cross-validation strategies for balanced and
9
imbalanced datasets, in: J. C. Xavier-Junior, R. A. Rios (Eds.), Intelligent Systems. BRACIS 2022.
Lecture Notes in Computer Science, volume 13653 of Lecture Notes in Computer Science, Springer,
Cham, 2022. doi:10.1007/978-3-031-21686-2_43.
[16] D. Chicco, N. Tötsch, G. Jurman, The Matthews correlation coefficient (MCC) is more reliable
than balanced accuracy, bookmaker informedness, and markedness in two-class confusion matrix
evaluation, BioData mining 14 (2021) 1–22.
[17] G. Da San Martino, Y. Seunghak, A. Barrón-Cedeno, R. Petrov, P. Nakov, et al., Fine-grained analysis
of propaganda in news article, in: Proceedings of the 2019 conference on empirical methods
in natural language processing and the 9th international joint conference on natural language
processing (EMNLP-IJCNLP), Association for Computational Linguistics, 2019, pp. 5636–5646.
[18] K.-H. Thung, C.-Y. Wee, A brief review on multi-task learning, Multimedia Tools and Applications
77 (2018) 29705–29725.
10